Systems and methods for detecting light sources

ABSTRACT

A method for detecting light sources, including capturing an image including a sub-infrared light emitter, applying a filter to a pixel of the captured image to isolate a signal strength of a range of frequencies, and comparing the signal strength of the filtered pixel to an expected signal strength of a background spectra for the range of frequencies. As a result of a difference between the signal strength of the filtered pixel and the expected signal strength exceeding a predetermined threshold, the method includes identifying the pixel as corresponding to a light emitter. As a result of the difference between the signal strength of the filtered pixel and the expected signal strength not a predetermined threshold, the method includes identifying the pixel as not corresponding to a light emitter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. application Ser. No.14/857,240, filed Sep. 17, 2015, and entitled “Systems and Methods forDetecting Light Sources,” which claims the benefit of U.S. ProvisionalPatent Application No. 62/051,732 filed Sep. 17, 2014, and entitled“Systems and Methods for Detecting Light Sources,” both of which areincorporated herein by reference in their entireties for all purposes.

BACKGROUND

Light emitting sources, such as light emitting diodes (LEDs) areincreasingly used for their efficiency and longevity relative toconventional light sources. In particular, in the aviation context, LEDsmay be used in an Approach Light System (ALS) or Medium IntensityApproach Lighting System with Runway Alignment Indicator Lights (MALSR)of an airport runway. However, unlike conventional light sources, LEDsused for lighting purposes, generate a spectrum in the visible range butdo not generate an IR spectrum component. Thus, conventional enhancedvision systems, such as those that rely on forward looking infrared(FLIR), which detects the IR component of light emitters, areineffective in detecting sources such as LEDs, whose spectrum iscontained only in the visible range.

BRIEF DESCRIPTION OF THE DRAWINGS

For a detailed description of exemplary embodiments of the disclosure,reference will now be made to the accompanying drawings in which:

FIG. 1 shows a system utilizing optical bandpass filters in accordancewith various embodiments of the present disclosure;

FIG. 2 shows a system utilizing tunable filter in accordance withvarious embodiments of the present disclosure;

FIG. 3 shows a plot of filter transmission bands of a tunable filter asa function of the filter angle in accordance with various embodiments ofthe present disclosure;

FIG. 4 shows a system utilizing a filter array and a focal plane arraycoinciding with the surface of a camera sensor in accordance withvarious embodiments of the present disclosure;

FIG. 5 shows an exploded view of a schematic of a filter array, alenslet array, and a sensor in accordance with various embodiments ofthe present disclosure;

FIG. 6 shows a plot of filter transmission spectra of bandpass filtersin the array of FIG. 5 in accordance with various embodiments of thepresent disclosure;

FIG. 7 shows a flow chart of a method for detecting light sources inaccordance with various embodiments of the present disclosure;

FIGS. 8-10 show emission spectrums of various colors of light emittersin accordance with various embodiments of the present disclosure.

FIG. 11 shows an enhanced overlaid image of several light emitters inaccordance with various embodiments of the present disclosure;

FIG. 12 shows exemplary ground reflection and atmospheric scatterspectra in accordance with various embodiments of the presentdisclosure;

FIG. 13a shows a visible portion of the background spectra normalizedfor atmospheric scatter and FIG. 13b shows an exemplary spectra for awhite LED in accordance with various embodiments of the presentdisclosure;

FIG. 14 shows sample frequency filter ranges in relation to the whiteLED spectra in accordance with various embodiments of the presentdisclosure;

FIG. 15 shows an exemplary combined visible spectra including bothbackground and LED components in accordance with various embodiments ofthe present disclosure;

FIG. 16 shows a signal strength plot of an output of the filters of FIG.14 for various pixels which correspond to the presence or absence of alight emitter in accordance with various embodiments of the presentdisclosure;

FIG. 17 shows a signal strength plot of an output of the filters of FIG.14 as a function of range in accordance with various embodiments of thepresent disclosure;

FIG. 18 shows a signal strength plot of an output of the filters of FIG.14 as a function of haze in accordance with various embodiments of thepresent disclosure; and

FIG. 19 shows an exemplary system block diagram detecting light sourcesin accordance with various embodiments of the present disclosure.

NOTATION AND NOMENCLATURE

Certain terms are used throughout the following description and claimsto refer to particular system components. As one skilled in the art willappreciate, companies may refer to a component by different names. Thisdocument does not intend to distinguish between components that differin name but not function. In the following discussion and in the claims,the terms “including” and “comprising” are used in an open-endedfashion, and thus should be interpreted to mean “including, but notlimited to . . . . ” Also, the term “couple” or “couples” is intended tomean either an indirect or direct electrical connection. Thus, if afirst device couples to a second device, that connection may be througha direct electrical connection, or through an indirect electricalconnection via other devices and connections.

DETAILED DESCRIPTION

The following discussion is directed to various embodiments of thedisclosure. Although one or more of these embodiments may be preferred,the embodiments disclosed should not be interpreted, or otherwise used,as limiting the scope of the disclosure, including the claims. Inaddition, one skilled in the art will understand that the followingdescription has broad application, and the discussion of any embodimentis meant only to be exemplary of that embodiment, and not intended tointimate that the scope of the disclosure, including the claims, islimited to that embodiment.

This disclosure is generally directed to vision enhancement throughturbid media, such as fog and cloud cover that occupy an operator'sfield-of-view. In particular, embodiments of the present disclosure maybe applicable in aviation fields or other areas in which enhanced visionis desirable, such as automotive or marine fields. The disclosed systemsand methods enhance the ability to perceive light emission from sourcesthat have no IR component, such as light emitting diode (LED) sourcesthat emit light exclusively in the visible range. Prior art enhancedvision devices, which are typically based on forward looking infrared(FLIR) technology, are only designed to detect the IR component of alight source, and thus are ineffective in detecting certain lightsources such as LEDs, whose spectrum is contained in the visible range.Throughout the present disclosure, reference will be made to LEDs forsimplicity; however, it should be understood that embodiments of thepresent disclosure may be equally applicable to other visible lightsources that have no or a minimal IR component, and thus cannot bedetected with conventional IR detection technology. These may also bereferred to as sub-infrared light emitters, which contain no or aminimal IR component. One feature of this disclosure is the detection oflight in the visible spectrum from a distance that exceeds the localvisibility range, particularly where this range is further limited byturbid media.

The present disclosure is directed to a camera system and a method fordetecting light sources that are obstructed by a turbid medium, evenwhere these sources would be undetectable by the human eye or byenhanced, contrast-based, non-spectral image processing, such as FLIR.In some embodiments, the system comprises a pixelated sensor, an opticalbandpass filter or set of filters, an imaging lens, and a processor. Thesensor, which is preceded in an optical path by the filter and theimaging lens, receives a set of images whose colors are defined by thebandpass filter(s). As a result, predominantly, a set of narrow,monochromatic images are rendered at the sensor, with varying signalcontent, which are subsequently transferred to the processor.

In accordance with certain embodiments, the processor executes analgorithm to process the image data by using the pixel coordinates andcolor, which reduces the level of background and clutter. The algorithmgenerates an image where light emitters such as LEDs become enhanced bya multi-spectral process, producing a visible image to the viewer,despite the fact that the LEDs are rendered invisible to the unaided eyeby the turbid medium set between the LEDs and the viewer. In accordancewith various embodiments, the optical filters employed in theabove-described system and method may span a broad range of theelectromagnetic spectrum, between the ultraviolet, through the visible,to the infrared regions.

In one particular embodiment, the above-described detection system isdeployed on board an aircraft, enhancing the pilot's ability to navigatein inclement weather and inferior visibility. In this case, the presentdisclosure may be referred to as one related to “instrument enhancednavigation” and may be included in the image presented to a heads updisplay (HUD) or heads down display (HDD).

Chromatic Filter

Rendition of a set of disparate monochromatic images can be accomplishedby several means. In one embodiment an array of optical bandpass filtersis disposed before the imaging lens, such that only one filter with asingle bandpass is positioned in the field-of-view at a given time, fora given camera shot. Then the filters are swapped. Following a cameratake, the filter array is moved, synchronously with the camera framerate, to a new position such that another filter with another singlebandpass is positioned in the field-of-view, and a new camera shot istaken. This process is repeated until camera takes with the entire setof filters have been made. In an embodiment the filters are placed in arotating filter-wheel, as shown in FIG. 1. Here the camera lens 102images rays originating, for example at infinity, on a focal plane 104,being a locus of the camera sensor. Further the filter-wheel, the crosssection of which is indicated as 106, holds optical bandpass filters,two of which are shown herein as 108 and 112. The wheel is rotatableabout the axis 110. In another embodiment, the filters are placed in alinearly translating filter strip. In yet another embodiment, thefilters are placed in a matrix with a two-dimensional translationcapability.

In a second embodiment, a tunable optical bandpass filter is disposedbefore or inside the imaging lens. The tunable filter may be one of anumber of such filters, for instance: a tilting, angularly movable,transmissive tunable filter; a liquid crystal tunable filter; anacousto-optical tunable filter; or a grating diffractive tunable filter.

FIG. 2 illustrates an embodiment employing a tilting, transmissivetunable filter 206. A camera lens 202 images rays (e.g., that may beconsidered to originate at infinity) on a focal plane 204, coincidingwith the surface of the camera sensor. Further the tunable filter 206 istilted by a rotating device 208. The center wavelength of the filterband is a function of the angle at which the tunable filter is set. Thusthe bandpass spectrum shifts on tilting the filter.

FIG. 3 shows a plot 300 of the bands as a function of the filter angle.The slight differences in the spectra for the s-polarization and thep-polarization, occurring at tilting angles greater than zero, providesa slight offset that can be accounted for in the selection of the bandsand their center wavelengths.

FIG. 4 shows an additional embodiment in which an array of opticalbandpass filters are disposed before the camera sensor. In the presentdisclosure, the filter mosaic contains optical bandpass filterscorresponding to various wavelengths, or bands, and arrayed in a twodimensional pattern. In certain embodiments, the bands may be arrayedcyclically at a certain period. For instance, if the bands are centeredaround the wavelengths of 450, 475, 500, 525, 550, 575, 600, 625, 650,675, 700, 725, 750, 775, 800 and 825 nm, the corresponding filters arearrayed in a square of 4×4 having four columns containing the followingbands: 1) 450, 475, 500, 525 nm, 2) 550, 575, 600, 625 nm, 3) 650, 675,700, 725 nm, and 4) 750, 775, 800 and 825 nm. The above example does notrepresent a limiting embodiment, and other arrays may be equivalentlyutilized. In some embodiments, no two filters of a like color areadjacent to one another.

In yet another embodiment, the array pitch and each elemental filter inthe array is equal to the pitch and size of the sensor pixel in a mannerthat once packaged together, each filter overlaps one pixel. In thisarrangement, various pixels become sensitive predominantly to a certaincolor, within the spectral range of the filter bandwidth. Illustrationsof this embodiment are shown in FIGS. 4-6. In FIG. 4, rays 400 (e.g.,that may be considered to originate at infinity) are imaged by a cameralens 402 through a filter array 404 on a focal plane 406 coinciding withthe surface of the camera sensor. From the polychromatic rays 400, onlythose colors (or wavelengths) transmitted by the individual filters ofthe array 404 arrive as predominantly monochromatic rays at the sensor406. FIG. 5 illustrates a blown-up schematic of the filter array andsensor unit 500, where polychromatic incoming visible light rays 502 areincident on a lenslet array 504. The visible light rays 502 pass throughan array of bandpass filters 506. The visible light rays 502 are thusfiltered and monochromatic rays 508 are predominantly transmitted, whichsubsequently arrive onto the sensor pixels 510. In one embodiment, thesensor pixels 510 are of a CMOS type sensor 512. FIG. 6 shows anexemplary plot 600 of transmission spectra of bandpass filters in thearray 506 shown in FIG. 5. Further, and as will be described in furtherdetail below, the exemplary bandpass filters of FIG. 6 correspond toexpected peaks and troughs of, for example, a white LED spectra, whichcan be utilized to further exemplify spectral regions in which the LEDis particularly distinguishable from a background spectra.

In accordance with embodiments of the present disclosure, known spectralstructures of the object light emitters are leveraged. The detailedspectra are expected to vary slightly from lamp to lamp. To addressthis, particular filter bands are selected to accommodate thisvariation. As such, it is not particularly important to have an exactspectral match, since all LEDs of a similar color are more or less proneto generate similar spectral patterns, even in the case of broadbandLEDs. For instance, white LEDs have a typical spectrum characterized bythe Correlated Color Temperature (CCT), which is specified for theobject lamp type, while color selections of LEDs are specified by theirwavelength range. Embodiments of the present disclosure allow foridentifying the presence of LEDs in a scene independent of theirspecific spectral shape. In particular, an analysis of the relativedifferences in spectra in the identified filter bands (e.g., thosecorresponding to known LED spectra) versus that of the spectralsignature of background scene is leveraged to identify the presence ofLEDs in a captured scene.

In accordance with certain embodiments, the background scene ischaracterized to provide known ground reflection coefficients as areference to compare against known atmospheric radiance spectrainclusive of down-welling radiance spectra. FIG. 12 shows an examplecharacterization 1200 of the average ground reflection spectra 1204 incomparison to average atmospheric scatter spectra 1202 as a function ofwavelength, including both the visible spectrum and wavelengths outsidethe visible spectrum. It can be seen that the atmospheric portion of thespectra 1202 is essentially a noise source and thus may be eliminated bynormalizing the observed image (e.g., subtracted from the measured datato produce a normalized ground reflection component, assuming theabsence of light emitters such as LEDs).

Embodiments of the present disclosure may utilize various methods, suchas the use of narrow band filters, to separate an individual LEDemission from an otherwise cluttered spectral background. In particular,reliance is made upon the improved signal to noise of a narrow portionof the spectrum (i.e., where the LED spectrum displays peak(s)) tobetter isolate the LED signal from the background. Certain embodimentsmay rely on signal averaging to do so to improve the realizable signalto noise ratio, although certain embodiments do not necessarily rely onthe presence of such averaging to discriminate between the LED signaland background spectra. One benefit of the disclosed systems and methodsis that eliminating the background signal completely is not required inorder to identify LED light sources.

Turning to FIG. 8, a fairly broad white LED spectrum 800 is shown forexemplary purposes. The white LED case demonstrates that a relativelysmall number of filters may be employed to capture the expected spectralpeaks of the LED relative to the background. Of course, it should beunderstood that where multiple LED types are expected (i.e., a moregeneral LED case), a larger set of filters may be employed to suitablycapture each expected LED type. For example, FIGS. 9 and 10 demonstratethat blue and red LEDs may generate peaks at different wavelengths,which could then be detected by centering a band on those particularwavelengths. The importance of the filter selection is to identify thoseregions in which the LED source is disparate from the background interms of its relative spectral content. The magnitude of the signal willalso depend upon the collection width of the system.

FIG. 13a demonstrates background reflection spectra 1300 that has, inthis case, been normalized to eliminate the contribution of atmosphericscatter 1202 shown in FIG. 12 and isolated in the visible range. As canbe seen, the background spectra 1300 is largely flat in the visibleportion of the spectrum particularly when discounting the variations dueto the atmosphere 1202 itself. As a result of this relative flatness inthe visible portion of the spectrum, varying selections of filters inthe visible range will provide nearly the same signal level for thebackground spectra regardless of particular filter values.

As can be seen in FIG. 13b , however, a white LED spectra 1310 containspronounced peaks 1312, 1314, 1316 in the visible range. These portions1312, 1314, 1316 of the white LED spectra 1310 provide a strong contrastwith the expected background 1300. In accordance with embodiments of thepresent disclosure, a selection of filter bands is carried out toleverage these areas of greater difference between LED spectra 1310 andan expected background spectra 1300.

FIG. 14 demonstrates an exemplary selection 1400 of six filter bands1402, 1404, 1406, 1408, 1410, and 1412. The band 1402 corresponds towavelengths of 415-435 nm; the band 1404 corresponds to wavelengths of440-460 nm; the band 1406 corresponds to wavelengths of 470-490 nm; theband 1408 corresponds to wavelengths of 550-570 nm; the band 1410corresponds to wavelengths of 605-625 nm; and the band 1412 correspondsto wavelengths of 750-770 nm. As should be appreciated, the selection1400 is for illustrative purposes, and the number and absolute placementof the filters shown is not intended to limit the scope of the presentdisclosure, including the claims. In fact, in certain embodiments theselection of the filter bands may be determined experimentally based onthe types of peaks and troughs expected for a particular illuminationsource.

In the exemplary selection 1400, three of the filters 1404, 1408, 1410have been chosen to correspond to the peaks of the LED spectra 1310while the other three 1402, 1406, 1412 have been chosen to correspond tothe troughs of the LED spectra 1310. In this manner, it becomes easierto distinguish which spectra are observed in FIG. 13. In the case whereLEDs are present in the captured image data, it is expected to see ahigher signal in bands 1404, 1408, 1410 than in bands 1402, 1406, 1412.However, in the case where a background scene is observed in capturedimage data, it is not expected to see an appreciable difference insignal strength across the bands 1402-1412.

FIGS. 13a and 13b also demonstrate that as the collection width of thesystem in angle, or even in spectral width, is opened, more signal iscollected. For example, the specific plots shown by 1300 a-d representbackground reflection spectra observed with increasingly widercollection widths. As the collection angle of the system or the spectralwidth of the filters is increased, a corresponding increase in thesignal that is collected on the sensor is observed. At the same time,however, the contrast of the LED spectra 1310 in relation to thebackground is reduced as in FIG. 13b . As with FIG. 13a , the specificplots shown by 1310 a-d represent LED spectra observed at increasinglywider collection widths. In particular, 1310 c and 1310 d are producedat such high collection width or spectral width of the filters that thepeaks found in the LED spectra are virtually eliminated.

As a result, it can be seen that narrow collection widths provide animproved contrast with respect to the background scene 1300 since theLED spectral differences dominate the captured image when nearer to thesource (i.e., LED), while increasingly wider widths begin to smooth outthe white LED spectra 1310 in relation to the background scene 1300.Unlike non-imaging narrow spectral band applications, however,embodiments of the present disclosure leverage both spectral propertiesas well as imaging properties of the disclosed optical system to exploitknowledge of various light emitters to more effectively filter thoseemitters from a background scene.

Embodiments of the present disclosure utilize a careful selection offilter values to be wide enough to collect enough energy to form theimage while remaining narrow enough to provide sufficient spectralresolution in the scene. Embodiments of the present disclosure alsoutilize a sensor portion of the system that is configured to resolve theindividual LED sources at the desired range, which of course may varybased on the particular application. In an aviation context, forexample, a desired range may be several kilometers ahead of theaircraft. Similar to band selection, the sensor portion also needs to bewide enough to collect enough energy while remaining narrow enough toprovide a clear distinction when an LED source is present in the scene.The sensor parameters may therefore be selected to take advantage of theselected band centers and spectral widths so the camera-controlparameters are capable of acquiring useful information from thealgorithm. By way of example and not limitation, in the case of aviationapplications, a sensor may provide at least a 30 deg×40 deg field ofview with sub-milliradian resolution in the scene to provide adequatesituational awareness. This selection is clearly dependent upon theparticular application of the system under consideration, so the scopeof the present disclosure, including the claims, is not limited to aparticular collection width of the either the optics or the spectralfilters. Similar methodologies may be applied regardless of the finalapplication.

Unlike traditional means of providing spectral discrimination, whereonly a narrow range of the spectrum is considered, a representativeimage of the scene can still be created from the overlay of the selectedbands. Unlike traditional means of imaging, a reliable indication of thepresence of LED sources can be distinguished from the background evenwhen the LED sources are not visible in the overall scene. This is moreclearly distinguished when considering the spectra at larger ranges inthe presence of atmospheric absorption and scatter effects. As oneexample, FIG. 15 shows measured sample spectra 1500 in the visibleregion based on the above-described example using six filters. Themeasured sample spectra 1500 is generated at a distance of one milebetween an LED source and an optical sensor, and considers the impact ofpath attenuation by drizzle and fog. The differing plots relate todifferent pixels of a captured image. As can be seen, there is no longerany clearly distinguishing presence of the LED spectra in the combinedcaptured image spectra. The image is also largely dominated by fog, sothe background scene itself is not distinguishable. In these conditions,embodiments of the present disclosure provide an overlay (e.g., on aHUD) of the LEDs 1102 in and otherwise degraded visual environment,resulting in an image similar to that shown in FIG. 11.

Referring back to FIGS. 12-14, the detailed spectra shown demonstratesthe magnitude of the spectral discrimination problem. These figures arenot intended to limit the scope of the disclosure to the particularembodiment of this disclosure that relies on a small number of filters,which have been selected to specifically identify individual sources inthe scene, such as LED lighting. Although each of the filter bands1402-1412 contains only a portion of the overall spectra (e.g., a subsetof the visual range) of the scene, each band 1402-1412 covers the sameobject scene; that is, the filter bands are applied to the same capturedimage. Their individual pixels can therefore be correlated back to thecaptured scene and overlap each other in the image. In this fashion,each band sees a similar portion of the scene, but at a differentwavelength. Thus, in some cases, individual scene pixels are queried todetermine whether bands 1404, 1408, 1410 are stronger than expected incomparison to bands 1402, 1406, 1412 to determine the presence of awhite LED source in an otherwise indistinguishable spectrum.

In accordance with embodiments of the present disclosure, the combinedvisible spectra 1500 shown in FIG. 15 is sampled by the selected filters1402-1412 shown in FIG. 14 to provide a sample filter selection output1600 shown in FIG. 16. The sample figure output 1600 in FIG. 16demonstrates the impact of the disclosed spectral discrimination systemsand methods. The sample output 1600 has been calibrated with respect tothe average background to provide a spectral comparison on a pixel bypixel basis in the image. As shown, cases 1-5 correspond to fivedifferent pixels of a particular captured image. Although the signalstrengths of pixels 1, 2, and 4 vary somewhat, these pixels are clearlydistinguished from pixels 3 and 5, which register no substantial peakswhatsoever. In accordance with various embodiments, it may be determinedthat pixels 1, 2, and 4 correspond to an LED source since the relativesignal in bands 1404, 1408, 1410 (as explained above, these bandscorrespond to the expected peaks of a white LED) is higher than in bands1402, 1406, 1412. At the same time, pixels 4 and 5 are indistinguishablefrom the expected background spectra. In accordance with certainembodiments, the results of such a pixel-by-pixel (or groups ofpixels-by-groups of pixels) determination may be stored in variousformats, such as a bitmap where a positive or true value for a pixeldenotes that pixel is associated with the presence of an LED against thebackground scene and a negative or false value for a pixel denotes thatpixel is not associated with the presence of an LED against thebackground scene. Of course, this is just one exemplary storage scheme.Subsequently, the bitmap or similar image may be overlaid to aconventional captured scene as is the case with a HUD for example,giving an operator (e.g. pilot of an aircraft) the ability to perceivethe LED sources despite the fact that those LEDs may not be visible tothe naked eye due to distance, atmospheric conditions, or a combinationthereof.

As shown above, the ability to distinguish individual LED sources fromthe background spectra depends on both filter selection and collectionangle. Further, the impact of range also should be considered, since anindividual pixel at larger ranges will inherently cover a larger portionof the scene than the same pixel will cover ranges closer to the LEDsource. Since the source is a fixed size, it will occupy an increasinglydiminishing portion of the pixel's signal as range is increased, similarto what is shown in FIG. 13. Embodiments of the present disclosureaddress this issue, which is a fundamental aspect of any imaging system.

For example, FIG. 17 demonstrates sample resulting filter output 1700where six exemplary filters described above are employed and a singlesource whose position is gradually moved further away from the systemoptics. The individual cases, labeled 1-9 on the plot, representincreasingly larger ranges from an LED source in comparison to theexpected limit of the background. As can be seen, and as with anyoptical imaging system, a fundamental limit exists on the maximum rangeat which an LED source can be distinguished from the background which.One of ordinary skill will appreciate this limit is a function of theoptical sensor design and may vary with application. However, dependingon the particular application, a bottom threshold may be set such thatpeaks below that threshold (e.g., detected peaks that are not more thana certain amount of spectral energy above the background) are notreported as positive identifications of a light emitting source. Inparticular, in the aviation field where safety and confidence are ofhigh importance, some bottom threshold may be established in which peaksbelow that are not likely enough to come from an LED source to reportthose peaks as such. For example, the difference in spectral energybetween the pixel analyzed in cases 8 or 9 and the background case maybe below the threshold, and thus neither cases 8 nor 9 are reported ascorresponding to an identification of a LED. Of course, in the case ofan airplane for example, as the plane proceeds toward the runway, thepeaks amplify as in cases 7 through 1, at which point the pixel inquestion may be determined to correspond to an identification of a LED.

In other words, LED sources can be distinguished in a consistently morereliable fashion as the range to the optical sensor is diminished. Athreshold of detection can therefore be selected which provides adesired signal to noise ratio from the background, which is capable ofidentifying individual sources in an otherwise indistinguishable scene,at all ranges closer than the limit defined by the selected threshold.That is, once the LEDs are identified, the strength of thatidentification will increase with diminishing range.

In degraded visual environments, a particular maximum range (i.e., wherethe difference in spectral energy between a light source and thebackground is below a predetermined threshold) may also be furtherlimited by apparent haze of the scene, which may be influenced byatmospheric turbulence in the intervening media. FIG. 18 demonstratesthe impact of increasing the atmospheric attenuation in a mannerconsistent with the absorption spectra of the atmosphere on a singlesource at a fixed distance; that is, FIG. 18 demonstrates the impact ofincreasing haze. The cases, labeled as 1-9 on the plot 1800, are againshown in increasing order, although rather than distance as in FIG. 17,a haze content of the atmosphere is increasing. As in FIG. 17, low hazeconditions provide a clear spectral contrast in bands 1404, 1406, 1410with respect to bands 1402, 1408, 1412, while increasing haze continuesto diminish this contrast until it is essentially indistinguishable fromthe relative background of the scene.

In some embodiments of the present disclosure, the systems and methodsmay be expanded to provide for enhanced capability relative to theabove-discussed example for distinguishing white LED sources from abackground scene by taking into consideration various atmosphericproperties or attributes. For example, when imaging dynamic sceneconditions, more information may be available on the expecteddistribution of the background scene image.

As one example, in an aviation application, the optical sensor array maybe divided into zones of designated scene content that address separatedata arrays, determined by the user or by the processor based on theexpectations of specific portions of the scene. Typical scene zonescould include a sky field-of-view that is predominantly detectingatmospheric radiance, a ground field-of-view predominantly detectingdown-welling radiance, and a field-of-view where light emitters arelikely to be found based on the position of the aircraft (e.g.,forward-facing for purposes of viewing emitters on a runway). In theparticular aircraft application, altitude and heading data may beutilized to determine where sky and landing lights are likely to befound.

The altitude and heading data may already be utilized to providesimulated scenes that are sometimes used to display ground informationon the HUD display at ranges where the actual scene would prove toonoisy for adequate display. As the aircraft comes closer to the ground,embodiments of the present disclosure may replace the simulated scenewith an actual image, including superimposed light emitters, for thepilot to make a determination of whether it is safe to land theaircraft. Although embodiments of the present disclosure are not limitedto the presentation of simulated scenes, the ability to separate thescene into expected zones based upon the position and heading of theaircraft is leveraged to determine which portions of the scene arelikely to be dominated by which zones when attempting to refine themethod of identifying light emitters in adverse conditions, particularlyin conditions where the scene may be dynamically changing.

In embodiments where dynamic scene changes are taken into account, theimage sensor still acquires image data and pixel counts of all thefilter bands are recorded, but separate analysis of scene content may beperformed to provide further fidelity in the calibration of thebackground content. For example, those zones seeing predominantlyatmospheric radiance may be used to provide further fidelity to theexpected absorption occurring in the sensor scene content. That is, thevalue of atmospheric radiance is generally known, and thus deductionsregarding absorption functions may be made by viewing those zones (e.g.,skyward) and determining differences to a known atmospheric radianceprofile. The determined absorption function may then be utilized to“correct” or provide additional fidelity to those zones seeingpredominantly radiance reflected from the ground, which improves thecapability to determine the relative contribution of absorption to abackground spectra in zones where light emitters are likely to be found.In certain embodiments, a variety of zones provide a similar ability toleverage known radiance values to determine atmospheric or otherconditional contributions to background spectra, improving the abilityto separate background spectra—including those atmospheric or otherconditional contributions—from the light emitters that are desired to belocated in the scene.

In another similar embodiment, atmospheric properties themselves can bemodeled at a higher fidelity when separating the scene in to zones. Inaddition to adjusting for expected absorption, a determination of theatmospheric radiance at the selected wavelength bands allows for areconstruction of the current atmospheric radiance spectrum to provide amore accurate representation of the down-welling radiance at theselected wavelength bands that reaches the ground. A reconstruction ofthe concurrent ground reflection radiance spectrum is then utilized totake into account the attenuation spectrum of the atmosphere in actualmeasured conditions. Again, in the context of enhanced aircraft vision,the foregoing may be accomplished by a device situated proximate to theairport and configured to transmit data to incoming aircraft regardingthe ascertained atmospheric radiance and concurrent ground reflectionradiance spectrum.

The expected spectra of the source(s) are still discriminated withrespect to the background, but uncertainties of the backgroundpixel-counts due to noise related to the incoherent combination of allambient and system contributions can be considerably reduced bycancelling out a more accurate representation of the backgroundradiance. The identification of the emitter location in sensor pixelcoordinates is a function of the pixel counts due to light emitters incomparison to the expected background pixel counts and, as a result, thediscrimination of the emitter radiance and spectra can be reliablyconducted at a considerably improved signal to noise when greaterinformation regarding the background radiance is known. The output isstill a processed image (e.g., a bitmap of emitter positions overlayinga captured image of the scene) with visible light emitters; however, theimproved signal to noise provided by the increased fidelity of theatmospheric properties allows that discrimination to be made atincreasing levels of distance or haze.

Referring to FIG. 7 and the above-described embodiments, a method 700for detecting a light source or emitter is shown. The method 700 beginsin block 702 with capturing an image including a sub-infrared lightemitter. As explained above, one benefit of embodiments of the presentdisclosure is the ability to detect light emitters that do not includean infrared component, even in scenarios where the emitters themselvesare obscured by a turbid media such as haze, or are at a distancegreater than what would be visible to the human eye. The method 700continues in block 704 with applying a filter to a pixel of the captureimage to isolate the signal strength of a range of frequencies. Asexplained above, the filter may comprise a bandpass filter, and therange of frequencies may correspond to an expected spectral peak ortrough of a particular light emitter, such as a white LED. It should beappreciated that although the method 700 relates to a single pixel,certain embodiments may repeat the method 700 for all or substantiallyall pixels in the capture image. Thus, in some embodiments, the imagecaptured may also be a set of predominantly monochrome images, whereeach image represents a certain color or wavelength (or range of colorsor wavelengths) of the scene. The set of images is digitized, using thepixel coordinates and color, and stored in a buffer, constituting whatmay be referred to herein as a “multi-spectral cube.”

The method 700 continues in block 706 with comparing the signal strengthof the filtered pixel (or pixel of a filtered predominantly monochromeimage) to an expected signal strength of background spectra for therange of frequencies. As explained above, a spectral signature ofbackground scene is largely known, as it depends from black bodyradiation of the sun compensated for atmospheric absorption, which canbe approximated as a noise component or determined experimentally, forexample based on observing a region having a known spectral distributionsuch as the sky, and correcting for added absorption of the atmosphere.Further, as demonstrated in FIGS. 12 and 13 a, the background spectra inthe visible range is largely flat and without large magnitude peaks,particularly relative to those demonstrated by a light emitter as inFIG. 13 b.

The method 700 continues in block 708 with determining or calculating adifference between the signal strength of the filtered pixel and theexpected signal strength of the background spectra for that frequency.If the difference in strengths is above a predetermined threshold (i.e.,the pixel demonstrates a higher signal strength than expected for abackground-only pixel), the pixel is identified as corresponding to alight emitter in block 710. If the difference in strengths is below thepredetermined threshold (i.e., the pixel demonstrates a signal strengthexpected for a background-only pixel, or close enough that it ispresents undue risk to identify as an LED, for example in an aircraftcontext), the pixel is identified as not corresponding to a lightemitter in block 712.

As described previously, in some cases the identification of whether apixel corresponds to a light emitter may be used to form a bitmap of thecaptured image in which pixels identified as corresponding to lightemitters are assigned a first value whereas pixels identified as notcorresponding to a light emitter are assigned a second value.Subsequently, processing on the captured image and bitmap may overlaythe bitmap on the captured image to generate an overlaid or enhancedimage, in which the background and clutter are substantially reduced,the signal of light emitters (e.g., LEDs) is augmented, and a processedimage is produced in which the light emitters are presented in theirtrue location in the sensor coordinates. These coordinates may beoverlaid with an existing scene image on the HUD or HDD, which may nototherwise provide a clear recognition of the light emitters. In theexample of aviation applications, this provides the pilot with a meansof distinguishing the runway lights 1102, which may be LEDs (or otheremitters having low IR spectral components), even in the presence ofdense fog, for example as illustrated in FIG. 11.

In some embodiments, the method 700 also includes applying multiplefilters to the pixel to isolate signal strengths for various ranges offrequencies corresponding to the multiple filters. The method 700 mayalso include analyzing a light emitter spectral radiance to identifypeaks and troughs for the purpose of experimentally determiningfrequency ranges where a bandpass filter would capture importantportions of the emitter signature. The usefulness of identifying peaksin the emitter spectra is explained above, particularly with respect toFIG. 14. However, identifying troughs in the spectra may also be usefulto provide a relative reference for peaks; in particular, since scenerycontent can vary dramatically depending on viewing conditions, therelative difference between an expected trough and an expected peak maybe leveraged to reduce instances of false positives or false negatives.

Turning to FIG. 19, a system is shown in accordance with variousembodiments. The system may be, for example, an enhanced vision system1900 for use in an aircraft or other application where augmenting anoperators perception of the surrounding environment is useful. Thesystem 1900 includes an image sensor 1902 coupled to a processor 1904and a memory 1906. The system 1900 may also optionally include a display1908, such as a HUD or HDD. As will be explained further below, thesystem 1900 may also be coupled to a ground-based image capture system1910, for example by way of a wireless or satellite link.

Similar to above, the image sensor 1902 is configured to capture animage including a sub-infrared light emitter. The captured image, aswell as other processed versions of the image (e.g., the set ofpredominantly monochrome images, where each image represents a certaincolor or wavelength, or the above-described multi-spectral cube) may bestored in the memory 1906. The processor 1904 is configured to receivethe captured image, for example from the sensor 1902 or memory 1906, andapply a filter (e.g., a bandpass filter) to a pixel of the capturedimage in order to isolate a signal strength component of a range offrequencies corresponding to that filter.

The processor 1904 is also configured to compare the signal strength ofthe filtered pixel to an expected signal strength of a backgroundspectra for the range of frequencies. If the difference in strengths isabove a predetermined threshold (i.e., the pixel demonstrates a highersignal strength than expected for a background-only pixel), theprocessor 1904 is configured to identify the pixel as corresponding to alight emitter. If the difference in strengths is below the predeterminedthreshold (i.e., the pixel demonstrates a signal strength expected for abackground-only pixel, or close enough that it is presents undue risk toidentify as an LED, for example in an aircraft context), the processor1904 is configured to identify the pixel as not corresponding to a lightemitter.

In some embodiments, the processor 1904 is configured to generate abitmap composed of identifications of whether the various pixelscorrespond to a light emitter, in which pixels identified ascorresponding to light emitters are assigned a first value whereaspixels identified as not corresponding to a light emitter are assigned asecond value. Subsequently, the processor 1904 may overlay the bitmap onthe captured image to generate an overlaid or enhanced image, in whichthe background and clutter are substantially reduced, the signal oflight emitters (e.g., LEDs) is augmented, and a processed image isproduced in which the light emitters are presented in their truelocation in the sensor coordinates. The processor 1904 may cause thedisplay 1908 to display these coordinates with an existing scene image,for example as a HUD or HDD, which may not otherwise provide a clearrecognition of the light emitters. In the example of aviationapplications, this provides the pilot with a means of distinguishing therunway lights 1102, which may be LEDs (or other emitters having low IRspectral components), even in the presence of dense fog, for example asillustrated in FIG. 11.

In other embodiments, the ground-based image capture system 1910 may beused to detect atmospheric properties at a higher fidelity than whenseparating a captured scene viewed by the image sensor 1902 in to zones,as described above. The captured image data from capture system 1910 maybe used to determine the atmospheric radiance at particular selectedwavelength bands, which allows for a reconstruction of the currentatmospheric radiance spectrum to provide a more accurate representationof the down-welling radiance at the selected wavelength bands thatreaches the ground. The processor 1904 may receive various informationfrom the capture system 1910; however, it should be appreciated that ingeneral, a reconstruction of the concurrent ground reflection radiancespectrum may be utilized to take into account the attenuation spectrumof the atmosphere in actual measured conditions. As one example, thecapture system 1910 is situated proximate to the airport and configuredto transmit data to incoming aircraft regarding the ascertainedatmospheric radiance and concurrent ground reflection radiance spectrum.Thus, in the context of enhanced aircraft vision, aircraft approachingsuch an airport will be provided with improved information regarding theatmospheric attenuation in actual conditions, or in a real time manner.

In some embodiments, the processor 1904 may also be configured to applymultiple filters to the pixel to isolate signal strengths for variousranges of frequencies corresponding to the multiple filters. Theprocessor 1904 may also be configured to analyze a light emitterspectral radiance to identify peaks and troughs for the purpose ofexperimentally determining frequency ranges where a bandpass filterwould capture important portions of the emitter signature. Theusefulness of identifying peaks in the emitter spectra is explainedabove, particularly with respect to FIG. 14. However, identifyingtroughs in the spectra may also be useful to provide a relativereference for peaks; in particular, since scenery content can varydramatically depending on viewing conditions, the relative differencebetween an expected trough and an expected peak may be leveraged toreduce instances of false positives or false negatives.

Sensor

In the above-described embodiments, the sensor receives a set ofmonochromatic images whose colors are defined by the bandpass filters.Its responsivity band is spectrally broad, predominantly spanning thebandpass region over which a substantial fraction of the incidentphotons that are absorbed generate electrons. Although the sensor ispredominantly insensitive to a particular color, or a narrow spectralslice of its range of responsivity, photon flux incident on the sensorpixels have certain colors, which are determined by the bandpass of thebandpass filters disposed between the sensor and the object (e.g., LED).In one embodiment, the monochromatic images are rendered in a temporalfashion, where during a single scan of the sensor area, pixel counts areobtained that correspond to a certain color defined by a bandpassfilter. A multi-spectral cube results from compiling the set ofpre-defined wavelength bands constitutes. Subsequently, additionalmulti-spectral cubes are rendered and transmitted, resulting in adynamic progression of images, or so-called footage.

In another embodiment, the plurality of monochromatic image renditionsis accomplished in a localized manner, where sub-pixel counts aregenerated by a mosaic-like, squarely-arrayed bandpass filter(s). In thisembodiment, in a single scan of the sensor area, pixel counts ofdisparate colors are available, defined by bandpass filterscorresponding to specific locations relative to the sensor coordinatespace, thus resulting in a multi-spectral cube. As above, additionalmulti-spectral cubes are rendered and transmitted, again resulting inthe generation of footage. Depending on the desired spectraldiscrimination characteristics, this may be performed on a variety ofspectral band divisions, such as a 3×3 pixel sub-array to provide for 9sensor array zones, a 4×4 pixel sub-array to provide for 16 sensor arrayzones, etc. This disclosure is not intended to be limited to a specificembodiment of the either the sensor array or its specific filterselections.

In order to enhance the detector sensitivity, such as the ability todetect and image light sources through a turbid medium such as fog, ahigh level of sensor acuteness may be required. Particular sensorparameters that may enhance the acuteness of the image include: broadspectral sensitivity; high quantum efficiency; high pixel resolution;low electronic noise (e.g., 1/f, generation-recombination, dark current,Johnson and readout noises); and large dynamic range or high A/Dconversion resolution. In certain embodiments, the sensor is a siliconbased CMOS type, sensitive over 400-1100 nm, having quantum efficiencyof 60% at 650 nm, having 2048×2048 pixels, readout noise of 1electron/pixel, and a 16-bit A/D conversion resolution.

Light Sources

One object of this disclosure is the detection of visible light sourcesthrough turbid obstructers. Light sources are typified by their level ofradiance, spectral emittance and etendue. These vary over very broadranges. For instance, black body emitters encompass the entireelectromagnetic spectrum, while lasers may have a very narrow wavelengthwidth, which can be on the order of picometers. Other sources, thoughdesigned for lighting in the visible range, still generate emission inthe IR range. In certain embodiments, although not limiting, theemitting sources are LEDs that emit predominantly in the visible range.

In an embodiment, the LED is a white emitting diode having an emissionspectrum represented by the plot in FIG. 8. In the double-humpstructure, the spectrum spans the range of 410-750 nm, all included inthe visible range, and has no components in the IR range. In anotherembodiment, the LED may be a blue light emitting diode having anemission spectrum represented by the plot in FIG. 9. It has a spectrumcentered at 468 nm and a FWHM width of 23 nm. In yet another embodiment,the LED may be a red light emitting diode having an emission spectrumrepresented by the plot in FIG. 10. It has a spectrum centered at 630 nmand a FWHM width of 17 nm. In a further embodiment the source is an LEDlamp used in an Approach Light System (ALS) or Medium Intensity ApproachLighting System with Runway Alignment Indicator Lights (MALSR) of anairport runway. There are additional light emitting sources effectivefor this disclosure, and none of the above mentioned embodiments shouldbe construed to place any limitations on the disclosure as a whole.

The Turbid Medium

Various turbid media in the optical path between the object light (e.g.,an LED) and the detector attenuate the source radiance and contributebackground radiance of their own. As a result, the object light isobscured to a viewer located proximate the detector, such as anaircraft, automobile, or boat pilot. The turbid media may includeenvironmental media such as fog, clouds, rain, hail, and snow. In thisdisclosure, the method enables the detection of obscured light sourcesfrom a distance exceeding the ambient visibility range. The termvisibility is defined as the range at which a light is detectable by thehuman eye. That in turn is inversely proportional to light attenuation.

In an embodiment, the detector can detect white LED with CCT (CorrelatedColor Temperature) of 3500K emitting 20 W into a cone of 10 degrees froma distance of 1160 meters where the ambient visibility is in the rangeof 350-800 meters. In another embodiment these parameters correspond tothe scenario of an aircraft landing at a descent angle of 3 degrees andan altitude of 200 feet, with fog permitting a visibility of 350-800meters RVR (Runway Visual Range) at the landing. In yet anotherembodiment the detector is mounted in an aircraft, providing image datato the pilot, which allows to the pilot to view an image of the LEDlight from an increased distance relative to the ambient visibility. Theactual ranges are a function of the system design and should not beconstrued to place limitations on the disclosure.

Embodiments of the present disclosure may also be directed to anon-transitory computer-readable medium. Such a computer-readable mediummay contain instructions that, when executed by a processor (e.g.,processor 1904), cause the processor to carry out all or portions of themethods and processes described herein.

The above discussion is meant to be illustrative of the principles andvarious embodiments of the present disclosure. Numerous variations andmodifications will become apparent to those skilled in the art once theabove disclosure is fully appreciated. For example, although referenceis often made to an airport-based or aircraft-based embodiment, thepresent disclosure may be employed on naval craft (and, for example, aport authority), automobiles, or other situations in which enhancedvision may be desired and where low-IR light sources are employed. It isintended that the following claims be interpreted to embrace all suchvariations and modifications.

What is claimed is:
 1. A method comprising: capturing an image includinga sub-infrared light emitter; applying a filter to a pixel of thecaptured image to isolate a signal strength of a range of frequencies;comparing the signal strength of the filtered pixel to an expectedsignal strength of a background spectra for the range of frequencies;and as a result of a difference between the signal strength of thefiltered pixel and the expected signal strength exceeding apredetermined threshold, identifying the pixel as corresponding to alight emitter; or as a result of the difference between the signalstrength of the filtered pixel and the expected signal strength not apredetermined threshold, identifying the pixel as not corresponding to alight emitter.
 2. The method of claim 1, further comprising: analyzing aspectral radiance of the light emitter and identifying a spectral peakof the light emitter; and utilizing a range of frequencies correspondingto the spectral peak in applying the filter.
 3. The method of claim 2,wherein more than one spectral peak is identified, the method furthercomprising applying a plurality of filters to the pixel, each of theplurality of filters configured to isolate a signal strength of a rangeof frequencies corresponding to one of the spectral peaks.
 4. The methodof claim 2, further comprising: analyzing a spectral radiance of thelight emitter and identifying a spectral trough of the light emitter;and applying a filter to the pixel of the captured image to isolate asignal strength of a range of frequencies corresponding to the spectraltrough.
 5. The method of claim 1, further comprising: repeating thesteps of applying, comparing, and identifying for each of a plurality ofpixels of the captured image; and generating a bitmap image whereinpixels corresponding to a light emitter are assigned a first value andpixels not corresponding to a light emitter are assigned a second value.6. The method of claim 5, further comprising: overlaying the bitmapimage on the captured image; and projecting the overlaid image as aheads-up display including indications of the light emitters.
 7. Asystem comprising: an image sensor configured to capture an imageincluding a sub-infrared light emitter; a memory configured to store thecaptured image; a processor coupled to the image sensor and the memory,the processor being configured to: receive the captured image and applya filter to a pixel of the captured image to isolate a signal strengthof a range of frequencies; compare the signal strength of the filteredpixel to an expected signal strength of a background spectra for therange of frequencies; and as a result of a difference between the signalstrength of the filtered pixel and the expected signal strengthexceeding a predetermined threshold, identify the pixel as correspondingto a light emitter; or as a result of the difference between the signalstrength of the filtered pixel and the expected signal strength not apredetermined threshold, identify the pixel as not corresponding to alight emitter.
 8. The system of claim 7, wherein the processor isfurther configured to: analyze a spectral radiance of the light emitterand identify a spectral peak of the light emitter; and utilize a rangeof frequencies corresponding to the spectral peak when the filter isapplied.
 9. The system of claim 8, wherein more than one spectral peakis identified, and the processor is further configured to apply aplurality of filters to the pixel, each of the plurality of filters isconfigured to isolate a signal strength of a range of frequenciescorresponding to one of the spectral peaks.
 10. The system of claim 8,wherein the processor is further configured to: analyze a spectralradiance of the light emitter and identifying a spectral trough of thelight emitter; and apply a filter to the pixel of the captured image toisolate a signal strength of a range of frequencies corresponding to thespectral trough.
 11. The system of claim 7, wherein the processor isfurther configured to: repeat the steps to apply, compare, and identifylight emitters for each of a plurality of pixels of the captured image;and generate a bitmap image wherein pixels corresponding to a lightemitter are assigned a first value and pixels not corresponding to alight emitter are assigned a second value.
 12. The system of claim 11,wherein the processor is further configured to: overlay the bitmap imageon the captured image; and cause a heads-up display to project theoverlaid image including indications of the light emitters.
 13. Anon-transitory computer readable medium comprising instructions that,when executed by a processor, cause the processor to: receive a capturedimage including a sub-infrared light emitter; apply a filter to a pixelof the captured image to isolate a signal strength of a range offrequencies; compare the signal strength of the filtered pixel to anexpected signal strength of a background spectra for the range offrequencies; and as a result of a difference between the signal strengthof the filtered pixel and the expected signal strength exceeding apredetermined threshold, identify the pixel as corresponding to a lightemitter; or as a result of the difference between the signal strength ofthe filtered pixel and the expected signal strength not a predeterminedthreshold, identify the pixel as not corresponding to a light emitter.14. The non-transitory computer readable medium of claim 13, wherein theinstructions further cause the processor to: analyze a spectral radianceof the light emitter and identify a spectral peak of the light emitter;and utilize a range of frequencies corresponding to the spectral peakwhen the filter is applied.
 15. The non-transitory computer readablemedium of claim 14, wherein the instructions further cause the processorto apply a plurality of filters to the pixel, each of the plurality offilters configured to isolate a signal strength of a range offrequencies corresponding to an identified spectral peak of a spectralradiance of the light emitter.
 16. The non-transitory computer readablemedium of claim 14, wherein the processor is further configured to:analyze a spectral radiance of the light emitter and identifying aspectral trough of the light emitter; and apply a filter to the pixel ofthe captured image to isolate a signal strength of a range offrequencies corresponding to the spectral trough.
 17. The non-transitorycomputer readable medium of claim 13, wherein the instructions furthercause the processor to: repeat the steps to apply, compare, and identifylight emitters for each of a plurality of pixels of the captured image;and generate a bitmap image wherein pixels corresponding to a lightemitter are assigned a first value and pixels not corresponding to alight emitter are assigned a second value.
 18. The non-transitorycomputer readable medium of claim 17, wherein the instructions furthercause the processor to: overlay the bitmap image on the captured image;and cause a heads-up display to project the overlaid image includingindications of the light emitters.